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(57) Abstract 



A distributed network management 
architecture, where a plurality of network 
managers share information about network 
elements by building the necessary infra- 
structure to discover alternate routes to ele- 
ment controllers when possible. A large 
number of network elements and opera- 
tion controllers or managed object agents 
spans are simultaneously accessible to mul- 
tiple graphical networic browser instances 
on physical workstations; Each network 
manager can manage an element controller 
directly, through a direct connection, or in- 
directly, through an indirect connection to 
a second network manager which directly 
manages that element controller. As well, 
a plurality of telecommunication networks 
can be federated for transparently increas- 
ing the number of users and the reliability 
of each netwoik. By allowing each net- 
work manager to be configured individu- 
ally, more flexibility over both engineering 
and survivability is achieved. 
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ARCHITECTURE FOR NETWORK MANAGER 
BACKGROUND OF THE INVENTION 

Field of the Invention 

The invention is directed to a network architecture and naore 
particularly to a distributed network management architectxu-e sharing 
information about network elements and providing increased 
survivability. 

Background Art 

Network management has become increasingly complex in 
today's telecommunications networks. Currendy, network managers 
directiy managed element controllers, which could be operation controllers 
(OPCs) or managed object agents (MOAs). Intelligent network elements 
(NE) are software driven in every aspect from maintenance to control, to 
release upgrades. . The management of these NEs requires a robust and 
highly efficient system which can process a large volume of data over a 
geographically distributed network. This highly efficient system must also 
provide network management tools for simplifying day to day operations 
and reduce service down-time. 

In the current network architecture, a network manager 
generally manages a maximum number of 75 element controller pairs, an 
element controller supports up to four NMs, and an OPC can control up to 
1200 network elements or networks. In addition, OPCs are limited 
processors that cannot handle more than four connections. 

As customer transmission networks grow, so does the demand 
for the number of users who need access to the system. As such, the 
number of NEs becomes larger and they are more geographically dispersed. 
All these changes cause significant technological challenges to the nature of 
network management. No longer can the entire customer network be 
managed centrally from a single point, rather the need for distributed 
network management, locally and geographically, is a grov^ng 
requirement. Additionally, customers wish to divide their network into 
different regions based on political or business boundaries. Quite 
frequently two or more regions overlap, presenting a challenge given the 
current engineering limits. 
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United States patent No. 5,375,199 (Harrow et aL issued on 
December 20, 1994 to Digital Equipment Corporation) discloses a method 
and device for monitoring performance of a computer system, including a 
graphical user interface (GUI). The GUI is designed to provide historical or 
real time information on the system and also allows the user to interact 
with the information being viewed. 

Uruted States Patent No. 5,261,044 (Dev et al. issued on 
November 9, 1993 to Cabletron Systems, Inc.) relates to a network 
management system which performs fault isolation. The information 
relating to the network is displayed, the network entities being represented 
by icons. The user may select a prescribed area of an icon to obtain details 
regarding a particular aspect of the network entity represented by the 

respective icon. 

However, these patents are not concerned with a distributed 
network management architecture designed for sharing information about 
the network elements and for allowing each NM to be re/ configured 
individually to manage the element controllers on a per span basis. 

SUMMARY OF THE INVENTION 

An object of this invention is to provide an architecture and 
core network manager changes that allow a large number of network 
elements and operation controllers (OPCs), or managed object agents 
(MO As) spans to be simultaneously accessible to multiple graphical 
network browser (GNB) instances on physical workstations. 

It is another object of this invention to provide an increased 
survivability of network to network manager workstation communication 
in case of network outages, by building the necessary infrastructure to 
discover alternate routes to controllers when possible. 

Accordingly, the invention comprises a method of managing an 
0 element controller of a teleconuntmication network using a plurality of 
federated network managers (NM), comprising the steps of connecting a 
first network manager (NM|) to the element controller (EC) for directly 
managing the EC, and connecting a second network manager (NM2) to the 
NMi for indirectty managing the EC. 
t5 The invention further comprises a method of federating a 

plurality of telecommunication networks for transparently increasing the - 
number of users and the reliability of each network, comprising the steps of 
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directly connecting a first network manager (NM^) to a first group of ECs, 
and connecting a second network manager (NM2) to a second group of ECs 
for direct management of the respective first and second group of ECs, 
providing at each ECs of the first group a collector name server vdth 
information on the NMi and any other NMs directly managing the first 
group, and providing at each EC of the second group a collector name 
server with information on the NM2 and any other MM directly managing 
the second group, establishing a connection between the NM2 and the 
NM|, upon a connection request from the NM2 to a first EC of the first 
group, establishing an indirect coimection between the NM2 and the first 
EC for indirect management of the first EC through the NM|, and upon a 
connection request from NM^ to a second EC of the second group, 
establishing an indirect connection between the MbAi and the second EC for 
indirect management of the second EC through the NM2. 

The invention also pertains to a method of federating a 
plurality of telecommunication networks for transparently increasing the 
number of users and the reliability of each network, comprising the steps of 
directly connecting a first network manager (NM|) to a first group of ECs, 
and connecting a second network manager (NM2) to a second group of ECs 
for direct management of the respective first and second group of ECs, 
providing additional direct connections between the NM| and selected ECs 
of the second group, and providing additional direct connections between 
the NM2 and selected ECs of the first group, and providing at each ECs of 
the first group a collector name server with information on the NM^ and 
any other NMs directiy managing the first group, and providing at the EC 
of the second group a collector name server with information on the NM2 
and any other NM directly managing the second group- 

Advantageously, NMs are configured to directly or indirectly 
manage element controllers on a per span basis. Indirect management 
allows the network manager user to specify the preferred method of 
managing the controller, reducing the total nimiber of direct connections to 
an OPC, while giving more network managers visibility of the network. 

Another advantage of the architecture according to the 
invention is self-healing, whereby a NM seeks other NMs to federate to, or 
promotes itself to direct management, when a NM directly managing an 
element controller fails. 
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Scalability and survivability together naean increased numbers 
of users in a variety of locations, managing a larger number of network 
elements transparently and with higher reliability. Some element 
controllers are managed directiy and even more, indirectly. Thus, the 
network architecture according to the invention allows management of up 
to ten thousand NEs, and up to seven hundred and fifty element 
controllers spans are simultaneously accessible. 

Still another advantage of the invention is that by allowing 
each network manager to be configxired individually, more flexibihty over 
both engineering and survivability is achieved. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, features and advantages of the 
invention will be apparent from the following more particular description 
of the preferred embodiments, as illustrated in the appended drawings, 
where: 

Figure 1 illustrates a network architecture with indirect and 
direct management of element controllers; 

Figures 2 shows the increase in user capacity aspect of scalability 
according to this invention; 

Figxures 3 illustrate survivability scenarios: Figure 3 A shows the 
points of coimectivity loss. Figure 3B shows how management of an 
element controller is effected by an alternative network manager (NM) in 
case of connectivity loss, and Figure 3C illustrates a reluctantly promoted 
management path; 

Figures 4A shows an exemplary network extending over two 

regions; 

Figure 4B shows a first phase in the network growth comprising 
a network operation centre; 

Figure 4C shows a second phase with an alternative network 
operation centre for survivability; 

Figures 5A illustrates an example of evolution of the network 
architecture of Figure 4A, using indirect and direct management; 

Figures 5B illustrates a further evolutionary stage of the 
network of Figure 5A using a federated connection between regional 
network managers, together with indirect and direct management; 
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Figure 6A shows the evolution of network of Figure 5B, with 
network operation centers, federated, direct and indirect connections; 

Figure 6B shows the architecture of Figure 6A, indirect 
connections removed; 
5 Figure 7 shows one possible reconfiguration of the network of 

Figiu-e 6A in case of failure of a regional network manager; 

Figure 8 shows how the number of users of configuration 
shown in Figure 6B may be increased using scalability; 

Figure 9 illustrates a further evolutionary stage for the network 
10 of Figure 6, designed for increased survivability; 

Figure 10 shows the main components of the network manager 
and the element controller for implementing the architecture of the 
invention; 

Figure 11 is a flow-chart for operation of the network to 
15 establish an indirect connection between a MM and an element controller; 
and 

Figure ,12 is flow-chart shovmig the operation of the network to 
re-establish a direct connection between a NM and an element controller, 

20 DESCMPTION OF THE PREFERRED ENfBODlMENT 

Definitions of some terms used in this specification are 
provided next, for a better understanding of the invention. 

A network manager consolidates the OAM&P information 
collected from the controllers in the network. An element controller, 
25 which could be an operation controller (OPC) or a managed object agent 
(MOA), manages a span, or sub-network of NEs, providing functions to, 
and collecting information from the NEs within the span. 

Scalability is defined herein as the ability to configure multiple 
network managers in such a way that they can share information about 
30 NEs, thereby increasing the number of NEs each network manager can 
manage without increasing the demands on the element controllers. 
Network managers configured in such a manner are called federated 
network managers, and as a whole, they are called a federation of network 
managers. 

35 Scalability introduces the concept of indirect management of 

element controllers via another network manager. Figure 1 illustrates the * 
principle of direct and indirect management according to this invention. A 
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first network manager 1 directly manages OPCs (operation controller) 2. 
This is shown by direct management path (connection) 3. A second 
network manager 4 manages OPC 2 indirectly, as shown by indirect 
management path (connection) 5, via a federated connection 6 to network 
manager 1. The direct connections are illustrated throughout the drawings 
in solid lines, indirect connections are illustrated in dotted lines, and the 
federated connections, in double lines, 

The number of element controllers that can be indirectly 
managed is larger than the number of element controllers that can be 
directly managed, thereby increasing the total number of network elements 
the network manager can manage. Conversely, there is no restriction on 
the number of network managers that indirectly manage an element 
controller. This is shovm in Figure 2, illustrating OPC 2 managed through 
direct management paths by four direct NMs 1, each direct NM 1 (D-NM) 
being connected to 24 indirect NMs 4. In this way, each indirect NM-4 (I- 
NM) can indirectiy manage OPC 2, resulting in a network architecture 
which allows management of up to ten thousand NEs, up to seven 
hundred and fifty element controllers spans being simultaneously 
accessible. 

It is to be noted that a network manager only acts as an indirect 
server for a controller in a federation, if and only if, it is directly connected 
to that controller. 

There is more to scalability than just sharing the information, 
scalability also provides a mechanism for increased survivability. 
Survivability is the ability for a federation of network managers to heal 
itself when there is a failure in the underljdng transmission control 
protocol/internet protocol (TCP/IP) layer or in one or more of the network 
managers. 

Figure 3A shows a simple network and two points of 
connectivity loss, for an exemplary network comprising NMs 1 and 8 
directly managing OPC 2 and a third NM 4, indirectly managing OPC 2 
through NMl, A first stuvivability scenarios disclosed in for an 
interruption of the management path 3 between OPC 2 and NM 1, a second 
scenario is disclosed for both an interruption of the management paths 3 
and management path 7 between OPC 2 and a NM 8, 

In the case of the 1st scenario, MN 4 cannot manage any more 
OPC 2 indirectly through NM 1, and it looks for an alternative 
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management path. As NM 4 is federated with NM 8 over federated 
connection 9, MM 4 detects this alternative NM 8 and re-establishes an 
indirect management path to OPC 2 through NM 8, as shown in Figure 3B, 

As indicated above, NM 4 has a preferred indirect connection to 
OPC 2. However, if both management paths 3 and 7 are interrupted, 
(second scenario), NM 4 cannot indirectly manage OPC 2, Now, NM 4 is 
forced to establish a direct connection because of the lack of an indirect 
connection. This connection is termed a direct reluctantly promoted 
connection. This connection/state is considered transitory, and NM 4 will 
attempt to restore back to indirect management at the first available 
opportuiuty. This scenario is illustrated in Figure 3C, where NM 4 
promoted itself to directly manage OPC 2 through a reluctandy promoted 
management path 10. 

Figures 4 A to 7 show examples on how a network evolves as 
the number of the subscribers grows and as the user's requirements change. 
These scenarios also depict one way the phased delivery of scalability 
' fimctionality could impact the evolution of network management in a 
company. 

Let's assume that the size of the initial network has recently 
grown such that a network provider (NP) can no longer manage the entire 
network from a single network manager. When this point was reached, 
the responsibility for the network is divided between two regions. Eastern 
and Western regions, as shown in Figure 6A. 

As well, let's assvune that the Western region is expected to 
grow more in the near future, so the provider plarmed ahead and deployed 
two network managers 30 an 40 at the Western region control center (RCC) 
50. 

Each network manager direcdy manages a number of operation 
controllers 31, 33, 35, 37, 39, 41, 43, 45, 47, and 49. The description and 
drawings refer to OPCs for simplification, but it is to be understood that the 
invention applies also to other element controllers. The OPCs are deployed 
as necessary at various sites in the Western and Eastern region. Figure 6A 
also shows that some OPCs are managed by two network managers, for 
reasons that will be explained later. Thus, OPCs 35 and 37 are monitored by 
both network manager 30 and network manager 40, OPCs 41 are monitored 
by network manager 40 in the Western region and network manager 60 in 
the Eastern region. 
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The explosive growth of the network of Figure 4A has led NP to 
realize that a network operations center (NOC) 70 need to be created for 
monitoring the entire network via direct connections 71, as shown in 
Figure 4B for the second phase of growth (NOC is also a NM), The regional 
5 network managers 30, 40 and 60 still exist for the day-to-day running of the 
network, while the NOC 70 is primarily used for off-hours support and 
demos to customers. 

As each controller can be directly managed by maximum two 
network managers in the phase shown in Figure 4A, NP realizes that the 

10 number of connections per OPC must be increased from two to four for 

allowing more connections per OPC and more OPCs per network manager, 
for some controllers, such as OPC 35, 37 and 41 which each need three 
connections to them. 

However, the solution of Figure 4B is temporary, for as soon as 

15 the number of OPCs in the network outstrips the maximum number that 
can be monitored from the network manager in the NOC, NOC 70 can no 
longer monitor the entire network. Increasing the number of element 
controller connections does nothing to enhance survivability, except 
increase the number of workstations that can monitor the network. 

20 In addition, the site for NOC 70 can be a dangerous place, subject 

to earthquakes, volcanoes, power outages and crime. NP is concerned 
about survivability of the network and has built an alternative NOC 80 at a 
different site, to take over in disaster scenarios, as shown in Figure 4C. 
Additionally, the alternative NOC 80 can be used to off-load NOC 70 during 

25 peak demand and holidays. Unfortunately, by providing alternative NOC 
80, provision of direct coimections 81 to the OPCs exhausts the current 
maximum number of connections (four) for OPCs 35, 37, and 41. 

Also, it is not possible to increase the number of the OPCs in the 
network of Figure 4B, since additional OPCs caimot be managed by NOC 70, 

30 the cxu-rent engineering limit being 150 OPCs for a NOC, or 75 OPC pairs. 
As well, if for some reason, more network managers were required to 
monitor controllers 35, 37, and 41, this would be possible only using 
indirect management. 

Not taking the NOCs into consideration just yet, indirect 

35 element controller management allows a reduction in the number of 

connections used at the sub-network controlling devices, as seen in Figure ' 
5A. The number of direct connections in group 51 and group 53 has been 
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reduced from four to three by implementation of indirect connections 63 
and 65. For implementing the indirect connections, a federated connection 
61 has been set between NM 30 and 40. No changes have been made to the 
OPCs that each network manager directly manages, NM 30, 40 and 60 see 
the same controllers as before indirect management. 

Next phase of growth provides for expanding the controllers 
managed by network manager 30 to include OPCs 37, 39 and 41, namely all 
OPCs in the Western region, as shown in Figure 5B. This was done by 
increasing the number of indirect connections in group 63 from one to 
three. Similarly, the OPCs managed by NM 40 include now all of the OPCs 
in the Western region, by increasing the number of indirect connections in 
group 65 from one to three. By indirectly connecting to OPCs 31, 33 and 35, 
more users have visibility of all of the NEs in the Western region. 

In the next phase of network growth, NOCs 70 and 80 are given 
a view of the entire network by federating each network operating center 
with the regional network managers 30, 40 and 60, as shown in Figure 6A. 
This implies providing federated connections 73, 74 and 76 for connecting 
NOC 70 with regional network managers 30 , 40, and 60, and federated 
connections 83, 84 and 86 respectively, for connecting NOC 80 with regional 
network managers 30, 40, and 60. Indirect connection 63 and 65 are 
configured as in the embodiment of Figure 5A. NOCs 70 and 80 do not 
directly manage any controllers, rather they manage all of the OPCs 
indirectly via the regional network managers. Implementing this is easy as 
it is simply a case of changing the existing direct connections to the OPCs, as 
shown in Figure 4C, into indirect connections. 

A much cleaner picture of the connections in the network is 
shown in Figure 6B, which is Figure 6A with the indirect connections 
removed. In reality, the indirect coimections are only logical connections, 
no surveillance or status information flows along these connections. 

Unfortxmately, it is not dear from Figure 6B the OPCs that each 
network manager indirectiy manages. This is not important from a users' 
point of view, but from an engineering point of view, this picture clearly 
depicts the "nailed up" connections between tire OPCs and the network 
managers, and the federated connections between the network managers. 

Regarding how survivable this configuration actually is, let's 
assume that, for example, NM 60 is accidentally turned off. Now, NOCs 70 
and 80 cannot indirectiy manage tiie OPCs 41, 43, 45, 47, and 49 in the 
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Eastern Region, and some form of direct management must be provided. 
One possible reconfiguration is shown in Figure 7, NM 60 with its direct 
connections is not shown for simplification and also because is not turned 
on. Controllers at 47 are now directly managed by NOC 70, and controllers 
5 at 43, 45 and 49 are directly managed by NCXZ 80. NCXZs 70 and 80 have 
created a federated connection 78 between them so that they can indirectly 
manage the OPCs the other is directly managing. 

In general, the sharing of the load of managing the OPCs must 
not necessarily be divided equally between the network managers. In the 

10 first phase of scalability, indirect connections should be attempted first to 
the network manager whose address is defined in a preference file, or 
lacking that, if sub-networks are implemented, to a workstation on the 
same sub-network as the indirect client. Either way, the number of 
simultaneous users of the NOC at each site has grown to exceed the 

15 maximum supported by a single workstation. However, using scalability, it 
is a simple case to add another workstation in the site of, for example NOC 
70, to allow more users, as shovm in Figure 8. 

All of the OPCs of Figure 8, but NM 41, now only have one 
connection to them, even though most of them have four network 

20 managers managing them. The addition of a third NOC 90, would not 

change these ntunbers. Additional connections to OPCs are now no longer 
required to get visibility from different network managers, rather the 
additional connections are used to engineer better survivability 
configurations or to restrict access for whatever reasons. 

25 Self-healing requires a NM to seek out other NMs to federate to, 

or in the worst case, a network manager has to resort to directly managing 
the controller. Finding another network manager to federate to is the ideal 
self-healing process, because the healing process is quite short, and 
therefore configuring the network to try and take advantage of this is 

30 highly desirable. One way to facilitate this objective is to have multiple 
network managers directly managing each OPC, so that federated network 
managers can quickly re-establish indirect connections. Eliminating single- 
points of failure is the primary goal. Ideally, the network managers directly 
managing each OPC should be connected via different routes and should be 

35 geographically separated to eliminate the chance of environmental 
problems disabling both network managers simultaneously. 
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The configuration of Figure 8 is still asymmetric in that the 
regional network managers still do not manage OPCs outside of their 
region. The resulting configuration is shown in Figtu-e 9. On the other 
hand, while adding multiple direct connections in one federation 
5 potentially increases the recovery time in the case of a failure (it is assumed 
that imtializing through a federated workstation will be quicker than 
connections to an OPC directiy), the total number of controllers that can be 
managed across the federation will be decreased, as the engineering limits 
will strictiy enforce the total number of direct connections per network 
10 manager. 

Taking all this into account NP decided to increase the 
responsibility of the NOC workstations so that they share in the 
management of the controllers, as shown in Figure 9. 

There are now multiple routes for a network manager to 
15 indirectiy manage an OPC- For example, network manager 70 could 

monitor OPCs 49 via federated connection 76 to network manager 60, and 
the respective direct connection of group 55, or via federated cormection 78 
to network manager 80, and direct connection 88. Both routes directiy 
manage OPCs 49. 

20 The responsibility of each regional network manager is wisely 

split in the configuration of Figure 9, so that \he effort is shared when a 
regional network manager is off-line. Thus, if network manager 70 were 
indirectiy managing OPC 49 via network manager 60 and network manager 
60 lost power, network manager 70 would switch to indirect management 

25 via network manager 80. If network manager 80 were indirectly managing 
OPC 41 via network manager 60, and network manager 60 lost power, 
network manager 80 would switch to indirect management via network 
manager 40. If network manager 70 were indirectiy managing OPC 45 via 
network manager 80, the failure of network manager 60 would have no 

30 impact. 

If subsequentiy network manager 40 failed, there would no 
longer be any network managers directiy managing the OPCs 41 and one of 
network managers 30, 70, or 80 would have to directiy manage OPCs at 41. 
For example, network manager 30 could reluctantiy promote itself to 
35 manage OPCs at 41, network managers 70 and 80 would then indirectiy 
manage this OPC via network manager 30. Similar recognition would 
occur for the other OPCs at 41 and the load of directiy managing these 
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orphaned controllers would be shared amongst the remaining network 
managers. 

If NP wishes to enter a joint alliance with a long-distance 
provider LD, they must share management of a set of OPCs, let's say OPCs 
5 41. Most probably, NP does not want LD to determine how the remaining 
of the network is managed. Conversely, LD does not want NP to determine 
how its network is managed. The solution is to configure separate 
networks to terminate at the OPC, and each company to configure their 
network manager to directly manage the shared OPCs. This does not 

10 change the NP configuration of the network of Figure 9 at all, except the 
nimxber of free connections on each of the shared OPCs is reduced. 

In the case of the multiple failure scenario above and shared 
OPCs at 41, when network manager 40 failed, network manager 30 would 
attempt to connect to the LD NOC 80. However, because the LD network 

15 manager is on another network, there is no IP connectivity to this network 
manager and network manager 30 is forced to resort to direct management 
of OPC 41, even though there is another network manager, namely NM 60, 
directly managing the OPC 41. Thus, configuring the IP network is 
important to prevent network manager 30 indirectly managing OPCs at 41 

20 via the LD network manager. 

Figure 10 shows the main components of a NM and an element 
controller relevant to this invention. Network manager 30 shown in 
Figure 10 is a workstation which provides nodal and sub-network OAM&P 
capabilities through a single view of transport and access nodes and directly 

25 manages a plmality of NEs that are connected to OPC 41, As indicated 

above, NM 30 may also indirectly manage a pliu-ality of OPCs and can act as 
a client or server for other NMs. 

NM 30 provides a graphical surveillance user interface (GUI) 
tool including a plxu-ality of graphical network browsers (GNB) 12 and a 

30 configuration GUI named the graphical network editor (GNE) 14. 

According to the invention, whether an element controller is directly or 
indirectly managed has no bearing on the performance or behaviour of the 
GNB 12 and is transparent to the GNB user. 

Network managers are configured by the user via GNE 14 to 

35 directly or indirectly manage element controllers on a per span basis. 
Provisioning what controllers the network manager is interested in by 
specifying their IP address, is done in the GNE 14 as usual. The 
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fundamental difference is that the operator may now select the preferred 
management path, direct or indirect. To this end, GNE 14 selects the 
element controller and the preferred management path, i.e. direct or 
indirect. An indirect server manager component (ISM) 34 and an indirect 
client manager component (ICM) 36 are the core of the scalability and 
survivability solution. These components are integrated into the existing 
network manager collector process and provide the ability to service 
incoming requests from other network managers as an indirect server, and 
to similarly establish connections to other network managers as indirect 
clients. 

ISM component 34 provides a mechanism to service indirect 
management requests from other network manager workstations. This is 
accompUshed by creating a TCP/IP server at a well known port for 
processing the incoming requests. 

The information flow between NM 30 as a directiy managing 
server and a client NM is provided over a federation network interface 
(NNI) 32. Once the interface is negotiated, the indirect server 34 waits for 
requests from the client NM for the specific OPC/MOA entities that the 
requester is interested in. Only requests for controllers that are currently 
being managed via a direct connection to the OPC or MOA such as OPC 41 
will be considered by ISM 34. 

When this negotiation is complete, information flow for the 
controller pair is tiransmitted from ISM 34 to tiie indirect client NM. Only 
one TCP/IP connection is maintained per communicating network 
managers. This information includes core network element information, 
such as name identifiers, rate, type and support information, existing active 
alarms, and asynchronously delivery alarm state transitions, performance 
monitoring support information, sub-network configuration data, and 
network element connectivity or association status. 

The role of ICM 36 is to find a path to a controller if the 
management path is specified by GNE 14 as indirect. It does this by first 
querying that OPC for the list of currentiy connected network managers, 
then attempting to register indirectiy through another network manager. If 
there are no direct managers available, ICM 36 will attempt to reluctantiy 
promote itself to direct management of that controller. 

When presented with a choice of two or more possible routes for 
indirect management, ICM 36 will choose a preferred route according to a 
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preference algorithm, if it is available. The preference algorithm uses an 
indirect access database (IAD) 38, and performs for example the following 
operations: 

First, IAD 38 is consulted by ICM 36 for the list of preferred paths. If 
5 there is a match, that path is attempted first. For example, an entry in IAD 
38 of form '47.105* means that presented two possible indirect paths 
•47.105,9.219" and '47.45.4.48*, '47.105.9.219' will be selected first. 

Next, if sub-networks are implemented, and if one of the routes is a 
network manager on the same sub-network as the client, that route is 
10 attempted first. 

Last, a random selection is made if no preferred path resulted from 
the above operations. 

This strategy allows for an additional level of control over 
indirect management, allowing optimal network paths (where there is 
15 more bandwidth, for example) to be established first, if available. 

IAD 38 is also consulted by ICM 36 before allowing access to an 
indirect manager client This database comprises of a list of IP addresses or 
sub-networks that requests will or will not be accepted from. This is used to 
facilitate the creation of independent federations which do not share 
20 management information or interact with each other. It is also used to 

ensure a predictable environment during software testing. The database 38 
is modifiable using a simple configuration tool accessible from the existing 
administration tools. 

The effect of increasing the total number of NEs impacts on the 
25 total amount of network configuration data that must be stored, including 
active alarms for the entire network. This requirement places additional 
burden on the MM physical resources. The imderlying assumption in 
evaluating the impact on NM, however, is that only a relatively small 
portion of that data will ever need to be duplicated through user interfaces 
30 at any one given instance in time. 

A network data cache (NDC) may be used in another 
embodiment of the invention for managing and storing data previously 
shared by all GNB's into a single cache entity, available for reading by all 
network manager processes. NDC is used to write various data in a shared 
35 area to be accessed in a read-only fashion by all other network manager 

processes. The goal of NDC is to achieve an order of magnitude savings as 
compared to the existing GNB 12 and GNE 14 process tmder similar loaded 
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conditions, thus minimizing or eliminating the requirement for additional 
hardware to achieve the engineering limits. 

A critical component of the architecture according to the 
invention is implemented in the collector name server (CNS) 46 
5 component provided at each OPC/MOA workstation. CNS 46, whose 

interface is implemented through the existing regulation browser (REGB) 
process on the OPC, provides a mechanism to query the OPC server for the 
existing directiy connected NM workstations, necessary for enabling ICM 36 
to select a path for indirect management. 
10 In the current design, access to the NE and OPC/MOA user 

interface is accomplished by using the primary network manager collector 
to controller interface to add entries in the appropriate control files on the 
controller, which then allows the network manager xmauthenticated 
remote execution of selected processes, if a concurrent entry for the 
15 invoking dient UNIX userid is present on the controller. 

As a design requirement, each indirect connection accoimts for 
the equivalent of two directiy managed element controller spans. This 
parameter is enforced and has been deduced by engineering studies on the 
existing architecture. An interface is provided by ISM 36, to manage and 
20 keep track of these indirect connections for the purpose of engineering the 
entire NM collector process. Specifically, because of the possibility of a 
failure of one or more indirectiy managing workstations causing the client 
to attempt direct connections to the controller and potentially well exceed 
its maximum number of direct connections, this number in conjunction 
25 with the current number of primary and backup direct cormections will be 
used to enforce engineering limits in the software. 

Additional changes should be made to the existing components 
of the NM. For example messaging optimizations between GNB 12 and the 
components of the NM must be provided as a result of these changes. The 
30 controller list dialogue should also be substantially modified to 

accommodate the now larger volume of controllers, as well as reflect the 
routing decisions being made by the network manager, such as IP address of 
network manager serving an indirect connection, indirect or direct 
preferred state, and the current actual management state of each controller. 
35 This information should also be visible to GNB users in a read only 
fashion. 
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It is suggested that Tl or equivalent bandwidth as a minimal 
requirement to sustain acceptable performance between federated network 
managers. Given the basic data provided, the entire network bandwidth 
can be engineered to accoimt for shifts in management responsibility that 
will happen when survivability aspects are considered. Any extra 
bandwidth required to maintain acceptable performance when shifts in 
management happen will largely depend on how the federation is setup 
and how many spans are being managed concurrently across the federated 
connections. 

Figure 11 illustrates how an indirect connection is established in 
the network of Figure 9. Let's suppose that NM 70 requests 
commtmication with OPC 41 as shovm in step 100. NM 70 queries OPC 41 
via CNS 46 regarding the NM directly connected to this OPC, in step 110. If 
a plurality of direct NMs is found, such as NM 40 and NM 60 in step 120, a 
preference algorithm selects a preferred direct NM in steps 130-150. 

When specifying a preferred direct connection, an attempt is 
made to register directly with the element controller 41 as usual. If both 
NM 40 and NM 60 failed, or none of NM 40 and NM 60 directly manage 
OPC 41, as illustrated along the NO branch in step 120, NM 70 would 
negotiate a self-promotion with the element controller to directly manage 
it, as shown in step 220. Information now flows between NM 70 and OPC 
41, step 230, and other network managers could then indirectly manage the 
element controller 41 via this promoted network manager. 

If the connection was established through a reluctantly 
promoted network manager, NM 70 will attempt to restore back to indirect 
management at the first available opportunity, as shown by the timer in 
step 240. 

As indicated earlier, if NM 40 and NM 60 are operational to 
directly manage OPC 41, NM 70 must select one of these two NMs (i=l or 
i=2) for indirectly managing OPC 41. The preference algorithm initiated in 
step 130 is designed to take into account the network architecture and also 
the clients demands. In steps 140 and 150 NM 70 deternunes which one of 
NM 60 and NM 40 has a preferred direct connection to OPC 41 and 
establishes a management path to this preferred NM. If a preferred direct 
connection is not available, as shown by branch 'NO* of block 150, NM 70 
determines which one of NM 60 and 40 has a preferred reluctantly 
promoted connection to OPC 41, as shown in steps 200 and 210. Step 160 
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shows the flow of information between NM 70 and OPC 41, once a 
management path has been established in steps 130, 140, 150, 200, and 210. 

Any changes in management path initiated under the control of 
the federation will attempt to happen transparent to the GNB user, in other 
5 words without indicating a loss of connectivity for those network elements 
affected (normally done by turning the affected icons blue). This is called a 
no-blue switch, 

A direct benefit of scalability is the inherent survivability that 
comes from the distributed management of the network. Survivability is 

10 implemented using a revertive N:l strategy that allows multiple network 
managers in the federation to maintain management of element 
controllers which can no longer be managed by another network manager 
for reasons such as workstation or network outages. This is accomplished 
by an indirect client, such as NM 70 in the example of Figure 9, seeking out 

15 another server, such as NM 60 in the federation, if its current server 

becomes unavailable, or if a controller becomes unavailable from the point 
of view of the current directiy managing server. The NEs operate as in the 
case of establishing an indirect coimection, shown in Figure 11. 

If a network manager workstation cannot directiy manage an 

20 element controller, it can no longer facilitate indirect management of the 
controller for federated network managers. Federated network managers 
that used to indirectiy manage that controller will now have to find 
alternate means of doing the management, directiy or indirectiy. As part of 
the self healing process, the affected network managers will first try to find 

25 another network manager directiy managing the NE controller and 

establish a federated connection to it for the purpose of indirectiy managing 
that NE controller. In the above example, NM 40 will be selected by the 
preference algorithm in step 130. 

Figures 12 is flow-chart showing the operation of the network to 

30 re-establish a direct connection. In step 300 NM 70 requests a direct 
cormection with OPC 47. If the registration to this OPC is full, i.e. ti\e 
element controller already serves four NMs, as shown by the 'YES' branch 
in step 310, the query is repeated at regular intervals of time, shown in step 
320, imtil OPC 47 becomes available, shown by branch 'NO*. In this case, 
35 information flows between NM 70 and OPC 47 along direct connection 77, 
as illustrated in step 330. 
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Any changes in management path that are initiated by a real 
loss of connectivity, such as an outage of the supporting indirect server 
workstation, or the lack of network cormectivity to an OPC or MOA 
workstation, will always be indicated to the user. 
5 If a network manager preferred management route is direct, and 

it cannot establish a direct cormection because the controller registration is 
full and all other connections are preferred direct, it will not demote itself 
to indirect. It is the responsibility of the federation manager to configure 
the network with no more than the maximum number of direct 

10 connections supported for that controller. 

Indirect management of an element controller does not mean 
IP connectivity is no longer required between the network manager and the 
element controller it manages indirectly. In fact, IP connectivity is required 
to set up indirect management in the first place. IP connectivity is also 

15 required for on-demand functionality such as remote login, PM query, 
remote inventory, shelf level graphics, and electronic software delivery. 
For increased survivability, IP cormectivity is required to allow the network 
manager to directly manage the element controller if for some reason it 
cannot find another network manager directly managing the specific 

20 device. 

Next, some survivability scenarios are discussed for the 
configuration of Figure 9. 

In the event of a loss of connectivity between an indirect MM, 
let's say NM 40 for OPC 33, and its client NM 80, the client will attempt to 
25 reconnect indirectly via another manager whose principally configured as a 
direct manager. This is accomplished by requesting the list of cormected 
network managers from CNS 62 of OPC 33, and posting a request to the 
preferred server. 

If there are no direct managers available, for example NM 30 is 
30 turned "off, NM 80 will attempt to reluctantly promote itself to direct 
management of OPC 33 if there are any connections available. 

The role of ICM 38 continues after recovery by continuously 
attempting to recover a connection via an indirect path. When a route 
becomes available, a switch over is completed without presenting a loss of 
35 connectivity to depending applications. By always attempting a return to 
this steady state, some level of control over communicating entities is 
preserved. 
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It is to be noted that this switch over may also impact other 
workstations in the federation, who may be managing indirectly via the 
reluctantly promoted path. The switch over is mandatory, although all 
attempts will be made to re-establish routes without presenting a loss of 
connectivity to the application. 

In the event of a direct MM, such as MM 30 losing cormectivity 
to an OPC/MOA, such as OPC 33, the indirect clients NM 70 and NM 80 
will be notified. A client, say NM 80 will then attempt to connect to 
OPC/MOA 33 and establish a connection via an alternate route, for 
example through directly connected NM 70, hence surviving a connectivity 
problem due to a network outage that only affected its current indirect 
server. If there is no TCP/IP connectivity between the indirect client 
workstation and the controller (i.e. the controller is down, or invisible due 
to the same network outage), it will continue to attempt indirect 
coimections until a path becomes available. 

In the event of a network manager outage, it is possible for 
more than one federated network manager to establish a direct reluctantly 
promoted connection to the same controller. Because this may afifect 
federation engineering, it is an xmdesirable state. 

To overcome this state, reluctantly promoted connections in the 
federation continuously attempt to demote themselves, although one 
reluctantiy promoted direct connection must always remain to ensure one 
server is available for the federation. 

Other specifications for the network managers that should be 
noted follow. 

If an optimal route is not available at the time of cormection 
resulting in the selection of another path, the ICM will not revert once the 
optimal path becomes available. 

Also of note is that multiple Ethernet interfaces are not 
supported; if a workstation is part of multiple sub-networks due to more 
than one LAN interface, the preferred path will only be discovered if this is 
true because of the first or default interface, as specified in the IP host map. 

When specifying indirect management for a span which 
contains both a primary and a backup controller, an attempt will be made to 
use the same indirect server for both the primary and backup controllers. 
Otherwise, the connection path for each controller is determined 
separately, which may result in the primary and backup getting their 
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management information from different indirect servers, or even both or 
one of the pair reluctantly promoting itself to satisfy its desire to find a path 
to that controller. 

The architecture allows for the future evolution of the 
5 messaging interface between the NM and the controller which will allow 
the indirect server to communicate addressing information about indirect 
clients to the controller, allowing the indirect clients to make login requests 
directly to the OPC or MOA, 

Because of the increased amount of data required to setup a 
10 large federation interested in most or all of each others directly managed 
spans, a tool could be provided to remotely retrieve the preferred directly 
managed spans of any network manager in the federation and apply any or 
all of those spans as preferred indirect in the local network manager's 
domain. 

15 Several areas of functional improvements are possible beyond 

the above description, as the previously mentioned load balancing when 
surviving a failure. The load on each NM is defined by the number of 
controllers it directly manages. This is configured manually by the user in 
the GNE and can be re-defined at any time, so that all NM directly manage 

20 their fair share of OPCs. 

In a true load balancing application, after a recovery has been 
completed, the network managers will communicate their ciurrent loads 
and hand-off direct management of element controllers to less heavily 
loaded workstations until a vmiform load is achieved. This hand-off 

25 should not impact functionality and should be transparent to GNB users. 

In indirect management there is no balancing, the load is 
distributed randomly based on which network manager is the first to claim 
a direct connection to each OPC, influenced by the network topology and 
the distribution of the network managers within that topology* It is 

30 anticipated that given the random nature of this algorithm, a uniform 
distribution will be achieved. 

While the invention has been described with reference to 
particular example embodiments, further modifications and 
improvements which will occur to those skilled in the art, may be made 

35 within the purview of the appended claims, without departing from the 
scope of the invention in its broader aspect. 
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WE CLAIM: 

1. A method of managing an element controller of a 

5 telecommunication network using a plurality of federated network 
managers (MM), comprising the steps of: 

(a) connecting a first network manager (NM|) to said element 

controller (EC) for directly managing said EC; and 

(b) connecting a second network manager (NM2) to said NM^ 

10 for indirecdy managing said EC. 

2. A method as claimed in claim 1, wherein step (b) comprises: 
providing at said EC a collector name server with information 

on all NMs direcdy managing said EC; 
15 providing at said NM^ an indirect server manager unit, and 

providing at said NM2 an indirect client manager; 

requesting at NM2 a connection with said EC, comprising 

querying said collector name server to locate a MM that directiy manages 

said EC and detecting NM^; 
20 establishing a federated connection between said NM2 acting as 

an indirect client for said NM^, and said NM^ acting as an indirect server 

for said NM2; and 

establishing signalling between said NM2 and said EC over an 
indirect connection through said NM|. 

3. A method as daimed in daim 1, further comprising the step 
of (c) connecting a third network manager NM3 to any of said NMi and 
NM2 for indirectly managing said EC* 

30 4. A method of federating a plurality of telecommimication 

networks for transparently increasing the number of users and the 

reliability of each said network, comprising the steps of: 

(d) directiy connecting a first network manager (NM^) to a first 

group of ECs, and direcdy connecting a second network manager (NM2) to a 
35 second group of ECs said NM^ and NM2 for direct management of said 

respective first and second group of ECs; 
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(e) providing at each ECs of said first group a collector name 
server with information on said NM| and any other NMs directly 

managing said first group, and providing at each EC of said second group a 
collector name server with information on said NM2 and any other NM 

directly managing said second group; 

(f) providing a federated connection between said NM2 and said 

NMi; 

(g) upon a connection request from said NM2 to a first EC of said 
first group, establishing an indirect connection between said NM2 and said 
first EC for indirect management of said first EC through said NM^; and 

(h) upon a connection request from said NM^ to a second EC of 
said second group, establishing an indirect connection between said NM^ 
and said second EC for indirect management of said second EC through said 
NM2. 

5. A method as claimed in claim 4, wherein step (g) comprises: 

(i) receiving said connection request from said NM2 to said first 
EC and detecting in said collector name server that NM| directly manages 

said first EC; 

(j) establishing a direct connection between said NM2 acting as 
an indirect client for said NM^, and said NM| acting as an indirect server 
for said NM2 over said federated connection; and 

(k) establishing an indirect connection between said NM2 and 
said first EC through said NMi- 

6. A method as claimed in claim 4, wherein step (h) comprises: 
(1) receiving said connection request from said NM^ to said 

second EC and detecting in said collector name server that NM2 directly 

manages said second EC; 

(m) establishing a direct connection between said NM| acting as 
an indirect client for said NM2, and said NM2 acting as an indirect server 
for said NM| over said federated connection; and 

(n) establishing an indirect connection between said NM| and 
said second EC through said NM2. 

7- A method as claimed in claim 4, further comprising the steps 

of: 
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providing additional direct connections between said NM^ and 

selected ECs of said second group; and 

providing additional direct connections between said NM2 and 

selected ECs of said first group. 

5 

8. A method as claimed in claim 5, wherein said step (j) 
comprises providing at said NM^ an indirect server manager unit for 
serving incoming requests from said NM2, and providing at said NM2 an 
indirect client manager unit for requesting indirect management of said 

10 first EC. 

9. A method as claimed in claim 6, wherein said step (m) 
comprises providing at said NM2 an indirect client manager unit for 

requesting indirect management of said first EC and an indirect server 
15 manager unit for serving incoming requests from said NM^. 

10- A method of federating a plurality of telecommunication 
networks for transparentiy increasing the number of users and the 
reliability of each said network, comprising the stieps of: 
20 directiy connecting a first network manager (NMi) to a first 

group of ECs, and connecting a second network manager (NM2) to a second 
group of ECs, said NMi and said NM2 for direct management of said 

respective first and second group of ECs; 

providing a federated connection between said NM| and said 

25 i NM2; 

providing additional direct connections between said NM^ and 

selected ECs of said second group, and providing additional direct 
connections between said NM2 and selected ECs of said first group; and 

providing at each ECs of said first group a collector name server 
30 v/ith information on said NMi and any other NMs directly managing said 
first group, and providing at each EC of said second group a collector name 
server with information on said NM2 and any other NM directly managing 

Sciid second group. 

35 11. A method as daimed in claim 10, further comprising 

providing at each of said NM| and said NM2 an indirect server manager 
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unit for serving incoming requests from other NM, and an indirect client 
manager imit for requesting indirect management of an EC. 

12. A method as claimed in claim 11, further comprising, lipon 
receiving a connection request from said NM| to said second EC applying a 

preference algorithm for establishing a preferred management path, and 
establishing a connection between said NM^ and said second EC over said 

preferred management path, 

13. A method as claimed in claim 12, wherein said preference 
algorithm comprises: 

querying said collector name server of said second EC to locate a 
directly managing NM, and detecting said NM2; 

determining if said NM2 is available for communication with 

said second EC; 

whenever said NM2 is available, 

establishing said preferred management path as an indirect 
coimection from said NM| to said second EC through said NM2; 

whenever said NM2 is not available, 

assuming a reluctantly promoted state at said NM| to directly 

manage said second EC, and establishing said preferred management 
path as a reluctantly promoted direct connection from said NM| to 

said second EC; and 

demoting said NM| from said reluctantly promoted state when 
said NM2 recovers, and establishing said preferred management path 
as said indirect connection; and 

continuously attempting to demote any NM from said 
reluctantly promoted state. 

14. A method as claimed in claim 11, further comprising 
providing a network operation center (NOC-NM) and directly connecting 
said NOC-NM with each of said NM| and MN2 over a respective federated 

connection. 

15. A method as claimed in claim 14, further comprising 
providing at said NOC-NM an indirect server manager unit for serving 
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incoming requests from said NMi and said NM2, and an indirect client 
manager unit for requesting indirect management of an EC. 

16. A method as claimed in claim 14, further comprising the 
steps of providing additional direct connections between said NOC-NM 
and selected ECs of said first and said second groups. 



17. A method as claimed in claim 16, further comprising, upon 
receiving a connection request from said NOC-NM to an EC, applying a 
preference algorithm for establishing a preferred management path, and 
establishing a coimection between said NOC-NM and said EC over said 
preferred management path. 

18. A method as claimed in claim 17, wherein said preference 

algorithm comprises: 

querying said collector name server of said EC to locate a directly 
managing NM and locating NMi; 

determirung if said NM^ is available for communication with 

said EC; 

whenever said NMj is available, 

establishing said preferred management path as an indirect 
connection from Said NOC-NM to said directly managing NM and to 
said EC; 

whenever said NM^ is not available, 

assuming a reluctantly promoted state at said NOC-NM to directly 
manage said EC, and establishing said preferred management path as 
a reluctantly promoted direct coimection from said NOC-NM to said 
EC; and 

demoting said NOC-NM from said reluctantly promoted state 
when said NMj recovers, and establishing said preferred 
management path as said indirect connection; and 

continuously attempting to demote any NM from said 
reluctantly promoted state. 
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19. A method as claimed in claim 11, further comprising 
connecting said NOC-NM with an alternative NOC-NM through a 
federated connection. 

20. A method as claimed in claim 19, further comprising the 
steps of providing additional direct coimections betv^een said alternative 
NOC-NM and selected ECs of said first and said second groups. 

21. A method as claimed in claim 20, further comprising; upon 
receiving a connection request from said alternative NOC-NM to an EC, 
applying said preference algorithm for establishing a preferred 
management path, and establishing a connection between said alternative 
NOC-NM and said EC over said preferred management path. 

22. A method as claimed in claim 17, wherein said preference 

algorithm comprises; 

querying said collector name server of said EC to locate a directiy 

managing NM and locating NMj and NM2; 

determining if any of said NMj and NM2 is available for 

commuiucation with said EC and selecting a provisioned favourite 

between said NMj and NM2; 

whenever said favourite NM is available, 

establishing said preferred management path as an indirect 
connection from said NOC-NM to said EC through said favourite 
NM; 

whenever said favourite NM is not available, 

assviming a reluctantiy promoted state at said NOC-NM to directiy 
manage said EC, and establishing said preferred management path as 
a reluctantly promoted direct connection from said NOC-NM to said 
EC; and 

demoting said NOC-NM from said reluctantiy promoted state 
when said favourite NM recovers, and establishing said preferred 
management path as said indirect connection. 

23. A method as claimed in claim 11, further comprising 
providing an indirect access database (IAD) at said NMi for storing the 



wo 98/59462 



PCT/CA98/00543 



- 27 - 

network addresses of all NMs which are indirect clients of said NM^, and 
the network addresses of all NMs which are indirect servers of said NM|. 

24. A method as claimed in claim 23, further comprising: 
protecting each network address in said IAD with an associated 

network provider ID; and 

configuring said telecommunication network for use by a 
pluraUty of network providers, according to said associated network 
provider ID. 
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